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A  high  speed,  teleoperated  unmanned  ground  vehicle  (UGV)  was  recently  demonstrated  in  United  States  Marine 
Corps  operational  exercises  at  Camp  Pendleton,  California.  Advanced  features  of  the  human-machine  interface  components 
used  in  the  remote  vision  systems  are  credited  with  allowing  both  novice  and  experienced  operators  to  effectively  perform 
UGV  tasks.  The  UGV  was  remotely  guided  across  terrain  ranging  from  smooth  asphalt  to  severe  undeveloped  land  at 
velocities  up  to  60  kilometers  per  hour.  Reconnaissance,  surveillance  and  target  acquisition  tasks  were  remotely  completed 
using  UGV  remote  vision  systems. 

Off-road  exercises  were  conducted  to  gain  an  understanding  of  how  UGV  ^stem  operation  is  affected  by  variations 
in  visual  display  system  features.  Observations  on  the  utili^  of  features  such  as  stereoscopic  vision,  color  imagery,  head- 
mounted  displays,  and  head-coupled  aiming  of  sensors  are  discussed.  Recommendations  are  provided  for  the  design  of  future 
UGV  remote  vision  qistems. 
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REMOTE  VISION  SYSTEMS  FOR 
TELEOPERATED  UNMANNED  GROUND  VEHICLES 

1  0  INTRODUCTION 

1  1  Background.  The  United  States  Department  of  Defense  (DoD)  supports  the 
development  of  Unmanned  Ground  Vehicle  (UGV)  systems  for  battlefield  applications. 
Three  versions  of  remotely  operated  UGV  systems  were  recently  demonstrated  during 
field  exercises  administered  by  the  DoD  UGV  Joint  Program  Office.  [1]  These  man-in- 
the-loop  systems,  called  "teleoperated"  systems,  remove  the  human  operator  from  the 
battlefield  and  offer  protection  from  hazards  encountered  by  the  remotely  controlled 
vehicle. 

The  Naval  Ocean  Systems  Center  (NOSC)  has  a  long  history  in  the  research  and 
development  of  remotely  controlled  vehicles  and  manipulator  systems  for  undersea,  land 
and  space  applications.  [2]  (3)  Ongoing  research  at  NOSC  focuses  on  teleoperated 
systems  designed  with  a  characteristic  referred  to  as  “remote  presence". 

"Remote  presence  is  the  perception  of  actually  existing  at  the  remote 
location.  The  degree  of  'presence'  achieved  is  determined  by  the  fidelity 
of  the  sensory  feedback  to  the  operator."  (2]. 

Based  on  NOSC's  experience  with  teleoperated  vehicles  and  research  in  rennote  presence 
principles,  remote  vision  systems  were  designed  to  support  development  of  the  TeleOperated 
Vehicle  (TOV)  system  -  a  United  States  Marine  Corps  (USMC)  sponsored  UGV  effort.  Lessons 
learned  from  this  development,  and  from  field  tests  of  TOV  vision  systems,  are  presented 
m  this  paper. 


Figure  1. 


TOV  Remote  Vehicle  (RV)  and  Control 


Station  (CS). 
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1.2  TeleOperated  Vehicle  fTOVI.  TOV  is  a  remotely  operated,  unmanned  ground  vehicle 
system  which  permits  an  operator  to  extend  sensory,  motor  function  and  problem 
solving  skills  to  a  vehicle  located  up  to  30  kilometers  away.  [4]  The  TOV  system 
consists  of  a  Remote  Vehicle  (RV)  and  a  Control  Station  (CS),  shown  in  figure  (1).  A 
human  operator  controls  the  TOV  system  using  displays  and  controls  located  at  the  CS. 
Visual  and  auditory  CS  displays  are  designed  to  facilitate  understanding  of  the  R^s 
situation.  Hand-  and  foot-operated  controls,  similar  to  automotive  vehicle  controls,  and 
head  position  sensors  provide  the  means  of  controlling  RV  functions.  A  fiber  optic  data 
link  allows  physical  separation  of  the  operator  from  hazards  encountered  by  the  RV.  The 
RV,  based  on  the  High  Mobility  Multi-Wheeled  Vehicle  (HMMWV),  Is  capable  of  high 
transit  speeds  and  can  traverse  severe  off-road  terrain.  Three  mission-specific  add-on 
subsystems,  called  mission  modules  are  accommodated.  Surveillance,  reconnaissance, 
and  target  acquisition  are  functions  which  are  currently  supported  by  these  mission 
modules. 

2.0  TOV  VISION  SYSTEMS 

2. 1  Overview.  The  operator  must  be  kept  abreast  of  changes  in  the  remote 
environment  to  effectively  control  the  TOV.  Since  vision  is  such  an  important  sense, 
faithful  portrayal  of  visible  surroundings  encountered  by  the  remotely  located  vehicle  is 
paramount.  A  UGV  operator  must  be  able  to  easily  interpret  visual  sensory  data  to 
perform  tasks  effectively. 

TOV  vision  systems  comprise  various  types  of  image  sensors  and  displays. 
Depending  on  the  task,  a  particular  subset  of  these  components  may  be  used.  TOV 
systems  make  use  of  two  separate  display  systems,  one  for  driving  the  remote  vehicle 
and  one  for  mission  specific  tasks  such  as  reconnaissance,  surveillance,  and  target 
acquisition  (RSTA). 

Remote  driving  is  compiicated  by  the  inherent  unpredictabie  nature  of  off-road 
terrain.  Fast,  accurate  scene  interpretation  is  needed  to  aiiow  operator  reaction  to 
terrain  conditions.  For  these  situations  a  head-mounted  stereoscopic  dispiay  is  used  in 
conjunction  with  head-coupled  video  sensors  on  the  vehicle. 

RSTA  tasks  typically  require  slow  panoramic  scans  of  the  sensors,  often  at 
deflection  angles  that  would  be  difficult  to  maintain  using  head-ooupled  control.  Panel 
mounted  video  monitors  are  used  in  these  situations  with  RSTA  sensor  pointing  direction 
controlled  with  a  joystick. 

2.2  Mobility  System.  Sensory  information  required  for  remote  driving  of  the  TOV  RV 
is  provided  to  the  operator  through  the  mobility  system.  The  mobility  system,  depicted 
in  figure  2,  is  comprised  of  the  mobility  sensor  platform,  located  on  the  RV  and  the 
HMD,  which  is  a  part  of  the  CS.  The  mobility  system  is  configured  with  visual  and  aural 
capabilities  that  give  the  operator  the  impression  of  being  present  at  the  remote  site. 
This  is  accomplished  by  sensing  the  remote  scene  using  a  stereoscopic  camera  pair 

and  binaural  audio  microphones.  The  pointing  direction  of  these  sensors  is  controlled 
with  operator  head  position  commands. 

RV  mobility  system  audio  and  video  information  are  presented  to  the  operator 
through  the  Control  Station  Head  Mounted  Dispiay.  A  head  orientation  sensor,  affixed  to 
the  operator's  helmet,  provides  position  commands  that  cause  the  mobility  sensor 


Figure  2.  TOV  Mobility  System  Block  Diagram. 


platform  to  track  the  operator  head's  yaw  and  pitch  orientation  with  one-to-one 
correspondence.  For  instance,  if  the  operator's  head  turns  to  the  left  30  degrees,  the 
mobility  sensor  platform  deflects  its  angle  to  the  left  by  30  degrees,  in  this  manner, 
sensors  located  on  the  remote  motorized  platform  are  aimed  using  head-coupled  control. 

2.3  Surveillance/Weapon  System.  Sensory  information  required  to  support  mission 
specific  tasks  is  support^  through  modular  additions  to  the  TOV  system.  Two  modules 
that  are  implemented  are  the  Reconnaissance,  Sunreillance  and  Target  Acquisition 
(RSTA)  and  weapon  modules.  Each  of  these  modules  has  azimuth/elevation  motorized 
mechanisms.  The  weapon  module  is  mounted  on  a  pedestal  and  has  the  capability  of 
slewing  an  M-2  50  caliber  machine  gun  through  a  range  of  positions.  The  RSTA  module 
has  a  sensor  suite  mounted  on  top  of  a  variable  height  extendable  mast.  There  is 
provision  to  slave  the  aim  of  the  weapon  module  to  the  RSTA  module  settings.  A  single 
joystick  is  used  to  control  the  pointing  direction  of  these  modules  individually  or  slaved 
together. 

The  weapon  module  has  a  single  video  camera  mounted  parallel  to  the  weapon 
barrel.  This  camera  is  used  to  supplement  safe  operating  procedures  by  providing 
visual  confirmation  of  weapon  aim  and  by  allowing  assessment  of  weapon  operation. 

RSTA  sensors  include  a  surveillance  targeting  camera  with  motorized  zoom  lens,  a 
forward  looking  infrared  (FUR)  camera,  and  a  laser  ranging/designator  camera.  These 
sensors  are  integrated  onto  the  RSTA  motorized  gimbal  platform. 

Panel  mounted  displays  provide  the  operator  with  mission  module  video  imagery. 
Depending  on  the  circumstances  of  the  mission,  the  operator  is  free  to  select  images 
from  any  two  of  the  RSTA  or  weapon  video  sensors  for  display. 


3.0 


TEST  AND  EYALUATlQtS 


3 . 1  Operational  Testing.  TOV  systems,  equipped  with  RSTA  and  weapon  modules,  were 
extensively  tested  with  U.  S.  Marine  Corps  personnel  during  a  series  of  operations  that 
assessed  the  viability  of  TOV  remote  control  concepts.  Five  phases  of  tests,  involving  two 
TOV  systems  and  numerous  civilian  and  U.  S.  Marine  Corps  operators,  allowed  NOSC 
engineers  to  experiment  with  alternate  vision  system  configurations.  During  these 
trials,  experts  in  the  field  of  UGV  vision  systems  were  invited  to  participate  in 
operational  tests  in  order  to  solicit  recommendations  for  system  improvement. 

Occasionally,  unscheduled  demonstrations  of  TOV  systems  occurred  during  the 
course  of  testing.  Individuals  without  TOV  experience  were  encouraged  to  participate  in 
limited  driving  runs.  Virtually  all  novice  operators  were  able  to  operate  the  TOV  after 
only  a  few  minutes  of  instruction.  Comments  obtained  from  these  drivers  and 
observations  of  their  experiences  provided  additional  information  to  system  designers 
and  are  discussed  in  this  paper. 

TOV  systems  were  successfully  driven  over  terrain  ranging  from  smooth  asphalt 
to  severe  undeveloped  land.  Remote  vehicle  velocities  in  excess  of  60  knVhr  were 
common  with  typical  driving  speeds  of  25  km/hr  over  dirt  and  gravel  roads. 

In  September  1989,  an  Advanced  Technology  Transition  Demonstration  featuring 
USMC  and  U.  S.  Army  teleoperated  systems  was  conducted  at  Camp  Pendleton,  California. 
[1]  This  exercise,  coordinated  by  the  UGV  Joint  Program  Office,  included  the  TOV 
system  and  the  U.  S.  Army's  Teleoperated  Mobile  All-Purpose  Platform  (TMAP). 
Experience  gained  while  preparing  for  and  conducting  demonstrations  confirmed 
previous  vision  system  observations. 

3.2  Human  Factors  Experiments.  NOSC  research  scientists  have  long  been  involved  in 
human  factors  .studies  related  to  UGV  and  other  teleoperated  work  systems.  [5][6][7] 
Current  research  efforts  focus  on  issues: 

1 )  comparing  stereoscopic  and  non-stereoscopic  vision  systems  for  driving  tasks; 

2)  comparing  driving  performance  of  HMDs  and  panel  mounted  display  options; 

3)  comparing  color  and  black-and-white  vision  systems  in  selected  UGV 

applications; 

4)  determining  the  best  compromise  between  video  sensor  field  of  view  and 

display  resolution  for  UGV  tasks. 

4.0  DISCUSSION  OF  REMOTE  VISION  SYSTEM  FEATURES 

4.1  Stereoscopic  Vision  refers  to  the  strong  sense  of  three-dimensional  depth 
perception  derived  from  retinal  disparity  cues  (differences  between  left-eye  and  right- 
eye  retinal  images,  arising  from  the  eye's  horizontal  separation).  Retinal  disparity  is 
only  one  of  many  cues  used  for  depth  perception,  but  it  is  one  of  the  strongest  and  least 
ambiguous  sources  of  information  about  relative  position  in  three-dimensional  space. 
The  visual  system  in  the  brain  analyzes  binocular  parallax  between  the  left  and  right 
retinal  images,  (along  with  many  other  visual  and  non-visual  cues)  to  infer  the  three- 
dimensional  location  of  surfaces  and  objects  in  the  scene. 

A  stereoscopic  video  sensor  and  display  system  typically  uses  two  cameras 
(separated  horizontally  by  the  nominal  human  eye  offset  of  ^  mm)  to  "stand  in*  for  the 
observer's  left  and  right  eyes,  picking  up  two  different  two-dimensional  images  that  are 
transmitted  by  the  display  system  to  the  observer's  own  left  and  right  retinas.  The 


visual  system  in  the  brain  then  treats  these  images  as  normal  visual  input,  as  if  the 
retinal  images  had  been  obtained  by  looking  directly  at  the  scene.  The  brain  uses  these 
two  retinal  images  as  information  sources  for  creating  a  three-dimensional  model  the 
scene. 

Studies  that  compare  stereoscopic  video  systems  with  monoscopic  video  systems 
for  remote  manipulation  tasks  revealed  the  positive  impact  of  stereoscopic  video  on 
operator  learning  and  In  poor  visibility  conditions.  I6){8]  From  these  studies  several 
conclusions  have  been  drawn.  Hightower,  et  al  [2]  report  that  stereoscopic  television 
displays  provide  substantial  performance  advantages  over  conventional  television 
displays  when: 

1)  aspects  of  the  remote  scenes  are  unfamiliar  or  are  frequently  changing; 

2)  the  rate  of  learning  of  new  tasks  is  important; 

3)  image  quality  is  poor; 

4)  tasks  have  significant  depth  positioning  requirements. 

Properly  implemented,  stereoscopic  video  systems  contribute  to  visual  realism, 
enhancing  the  operator's  sense  of  remote  presence  at  the  teleoperated  vehicle's  location. 

TOV  is  designed  with  stereoscopic  vision  for  vehicle  mobility  operations.  This 
approach  Is  justifiable  when  one  considers  the  difficulty  of  maneuvering  a  remote 
vehicle  at  speeds  up  to  80  kilometers  per  hour  through  unfamiliar  off-road  terrain.  It 
is  essential  that  the  operator  have  the  means  to  detect  obstacles  and  terrain  features 
which  may  prevent  safe  vehicle  passage.  In  addition,  the  RV  may  encounter  poor 
visibility  conditions  caused  by  smoke,  fog,  dust,  vegetation,  and  low  light  illumination. 

In  these  situations,  stereoscopic  depth  perception  provides  additional  visual  cues  to  the 
human  operator  and  enhances  the  operator's  chances  of  successfully  completing  driving 
tasks. 

TOV  has  two  miniature,  high  resolution  video  cameras  that  provide  stereoscopic 
image  sensing.  These  cameras  are  aligned  with  parallel  optical  axes  and  are  separated  a 
distance  equivalent  to  an  average  human's  interocular  separation.  The  video  camera  lens 
field  of  view  (FOV)  is  selected  to  match  the  apparent  disf^ay  FOV  observed  by  the 
operator.  Automatic  iris  lenses  allow  unattended  aperture  adjustment. 

An  HMD  is  used  as  the  TOV  stereoscopic  di^lay.  The  HMD  provides  separate  left- 
and  right-eye  displays  corresponding  to  the  left  and  right  video  cameras.  Since  the 
apparent  HMD  FOV  nearly  matches  the  video  camera/lens  FOV,  an  image  magnification  of 
approximately  1.0  is  realized. 

Camera  convergence  distance  was  selected  through  experimentation.  Initial 
remote  driving  tests  were  perfonned  with  video  cameras  aligned  for  convergence  8 
meters  in  front  of  the  vehicle's  hood.  The  rationale  for  this  alignment  was  that  such  a 
configuration  would  enhance  depth  perception  in  the  immediate  vicinity  of  the  RV. 
However,  in  prolonged  instances  of  remote  driving,  operators  reported  perceptual 
discomfort  when  viewing  distant  objects  on  the  horizon.  This  may  be  explained  by  the 
need  to  diverge  the  eyes,  beyond  parallel,  in  order  to  fixate  objects  further  than  8 
meters.  Since  eye  divergence  is  not  normal,  operator  fatigue  results  due  to  eye  strain. 

Fair  subsequent  tests,  the  camera  optical  axes  were  aligned  parallel  to  each  other 
(converged  at  infinity).  Eye  convergence,  when  using  the  HMD  in  this  configuration,  is 
nearly  identical  to  eye  convergence  when  viewing  the  remote  scene  directly.  Eye 
divergence  is  not  required  to  fuse  any  of  the  objects  within  view.  There  were  no  further 
reports  of  viewing  discomfort  following  this  adjustment. 

The  stereoscopic  effect  is  important  to  remote  driving,  especially  near  the 
vehicle.  This  area  is  of  immediate  concern  with  regard  to  obstacles  and  terrain  features. 


RV  remote  driving  trials  were  recorded  using  videotape  and  still  photographs.  Analyses 
of  these  recordings  revealed  that  objects,  such  as  rocks  buried  in  vegetation,  and 
negative  terrain  features  such  as  potholes  and  ditches,  were  difficult  and  sometimes 
impossible  to  discern  using  monosoopic  vision.  Stereoscopic  views  of  the  same  scene 
increased  the  detectability  of  these  hazards. 

4.2  Color  Imagery.  TOV  remote  vision  systems  were  initially  designed  to  support  black- 
and-white  sensors  and  displays.  Several  factors  influenced  this  decision  to  exclude  color 
imagery.  First,  there  were  no  definitive  references  that  favored  color  images  for 
remote  vehicle  driving  and  tactical  missions.  Further,  the  high  cost  and  increased 
complexity  to  support  color  imagery  in  low  light  illumination  precluded  its  use  at  night. 
Finally,  the  technology  to  support  color  head-mounted  displays  for  driving  were  not 
adequately  developed. 

Despite  the  original  decision  not  to  include  color  cameras  and  color  CRT  monitors, 
NOSC  engineers  designed  TOV  telemetry  electronics  to  be  compatible  with  color  video 
(NTSC)  standards.  This  permitted  TOV  system  experimentation  with  a  color 
surveillance  zoom  camera  and  color  panel  mounted  display.  Anecdotal  data  from  TOV 
operators  indicate  that  in  certain  RSTA  situations,  color  improved  operator  recognition 
of  targets.  However  in  poor  lighting  situations,  sudi  as  near  dusk  and  in  shadows, 
recognition  of  targets  was  degraded  due  to  color  sensor  limitations. 

Videotape  segments  of  TOV  driving  exercises  and  segments  using  hand-held  video 
camcorders  were  compiled  to  observe  whether  color  imagery  contributes  to  effective 
remote  driving.  Anecdotal  data  indicate  that  color  rendition  of  certain  terrain  features 
enhances  operator  recognition  of  those  features.  For  example,  dirt  roads  lined  with 
vegetation  stood  out  in  color  because  of  the  contrast  between  the  brown  path  and  green 
borders.  Wet  and  burned  vegetation  were  more  noticeable  when  viewed  in  color.  Rocks 
hidden  in  tall  grass  blended  in  with  shadows  when  viewed  in  black-and-white,  but  were 
recognizable  as  distinct  objects  when  color  was  added. 

Practical  Implementation  of  color  vision  systems  is  limited  by  technology.  Low 
light  level  video  cameras  are  not  inexpensively  available  to  support  color.  In  addition, 
color  HMD  technology  is  not  readily  available  to  support  a  full  color,  moderate 
resolution  and  field  of  view  display.  System  complexity  is  increased  by  the  need  to 
sense,  transmit  and  display  color  images.  It  is  clear  that  further  experimentation  is 
needed  to  assess  the  performance  tradeoffs  associated  with  color/non-color  sensor  and 
display  possibilities. 

4.3  Terrain  Slope  Perception.  Operator  perception  of  remote  vehicle  attitude  is 
essential  to  complete  certain  UGV  driving  maneuvers.  Pitch  and  roll  attitude  cues 
provide  operators  with  an  understanding  of  vehicle  state  and  terrain  grade/slope.  This 
data,  combined  with  knowledge  of  vehicle  driving  characteristics,  allows  the  operator  to 
dedde  whether  transit  through  encountered  terrain  is  advisable  or  even  possible. 

Without  ^xoper  attitude  awareness,  the  operator  lacks  crucial  information  needed  to 
effectively  negotiate  off-road  terrain.  For  example,  if  the  operator  fails  to  recognize 
severe  terrain  slopes,  the  remote  vehicle  may  be  driven  inappropriately  on  steep  hills, 
resulting  in  vehicle  roll-over. 

TOV  system  drivers  gain  an  understanding  of  remote  vehicle  pitch  and  roll  attitude 
through  observation  of  HMD  video  information.  Pitch  and  roll  alphanumeric  indicators 
are  also  accessible  but  are  not  normally  used  while  driving.  The  mobility  sensor 
platform  is  positioned  on  the  vehicle  so  that  the  remote  vehicle's  hood  is  normally  in 


view  at  the  bottom  of  the  display  screen.  Since  the  mobility  sensor  platform's  roll  axis 
is  fixed,  relative  to  the  vehicle,  the  vehicle's  hood  is  normally  displayed  with  its  top  edge 
horizontal  to  the  display  screen.  When  the  vehicle's  attitude  is  affected  by  terrain,  the 
change  in  vehicle  attitude  is  portrayed  through  changes  in  displayed  terrain  features. 

For  example,  if  the  vehicle  is  on  a  side  slope,  then  trees  that  are  displayed  in  the  video 
monitors  will  tilt  to  one  side.  In  contrast,  visible  parts  of  the  remote  vehicle  retain  a 
fixed  attitude  in  the  display.  This  frame  of  reference  may  be  termed  "vehicle 
referenced"  since  the  cameras  are  referenced  to  the  vehicle's  normal  axis,  and  the 
displayed  vehicle  attitude  remains  constant. 

When  viewing  vehicle  referenced  displays,  it  is  easy  for  operators  to  mistakenly 
perceive  the  vehicle  to  be  oriented  upright,  regardless  of  the  actual  orientation.  Unless 
the  operator  has  unambiguous  visible  attitude  references,  such  as  landmarks  or  the 
horizon,  disorientation  can  result.  This  is  especially  true  under  degraded  viewing 
conditions.  If  the  operator  is  incapable  of  seeing  more  than  a  few  meters  past  the 
vehicle,  it  would  be  difficult  to  interpret  vehicle  attitude. 

During  TOV  operations  and  tests,  operators  occasionally  experienced  a  distorted 
sense  of  RV  attitude.  In  these  situations,  they  were  not  able  to  understand  RV  pitch/roll 
attitude  without  protracted  visual  analysis  of  the  display.  During  dynamic  driving 
conditions,  when  vehicle  attitude  is  constantly  changing,  the  lack  of  adequate  attitude 
awareness  may  lead  to  dangerous  driving  situations. 

Real-life  examples  highlight  the  operator's  difficulty  to  recognize  RV  pitch  and 
roll  attitude.  While  conducting  tests,  the  RV  was  occasionally  driven  and  parked  on 
terrain  side  slopes.  In  several  instances  when  normally  upright  objects,  such  as  utility 
poles  or  test  personnel,  came  into  view,  operators  were  surprised  to  see  them  standing 
at  a  severe  angle.  Upon  further  pondering,  operators  reported  that  they  were  forced  to 
accept  the  fact  that  the  objects  wgre  vertical  and  fftat  the  RV  was  on  an  incline,  if  UGV 
operators  are  so  easily  misled,  hazardous  terrain  conditions  may  be  incorrectly 
interpreted  and  may  threaten  the  effectiveness  of  the  UGV  system. 

To  study  this  effect,  NOSC  scientists  initiated  a  series  of  experiments  and 
considered  possible  remedies.  Of  particular  interest  are  methods  that  make  the  operator 
intuitively  aware  of  remote  vehicle  attitude.  Promising  approaches  include  the 
superposition  of  graphical  symbology  to  the  driving  video  display,  motion  simulation 
feedback  to  the  operator's  seat,  and  the  use  of  "gravity  referent"  compensation  to  video 
sensors. 

Gravity  referenced  compensation  keeps  the  video  sensors  at  a  constant  pitch  and 
roll  orientation  with  respect  to  the  earth's  gravity.  The  mechanism  maintains  its 
gravity  reference  independent  of  the  vehicle's  attitude.  As  a  result,  the  image  of  the 
vehicle's  hood  is  obsenred  to  pitch  and  roll  as  the  vehicle's  attitude  is  affected  by  terrain 
conditions.  This  tends  to  serve  as  a  natural  visual  indicator  of  vehicle  attitude. 

Videotape  simulations  using  video  cameras  demonstrate  the  advantage  of  the 
gravity  referenced  sensor  principle.  Scenes  were  recorded  using  a  video  camcorder 
located  near  the  vehicle's  driving  cameras.  The  vehicle  was  driven  through  a  course 
with  severe  grade  and  side  slope.  When  taping  the  vehicle  referenced  version,  the 
camera  was  held  against  the  vehicle  roll  bar  to  maintain  a  fixed  camera  view  relative  to 
the  vehicle.  During  gravity  referenced  trials,  the  hand-held  camcorder  was 
continuously  adjusted  to  maintain  a  gravity  referenced  position. 

It  was  observed  that  the  gravity  referenced  view  of  the  course  provided  more 
readily  interpretable  vehicle  pitch  and  roll  attitude  cues.  The  changing  vertical  position 
of  the  vehicle's  hood  within  the  display,  provided  an  indication  of  changing  terrain  grade. 


Similarly,  terrain  side  slopes  were  perceived  through  vehicle  hood  tilt  angles  across  the 
display  screen.  In  addition  to  providing  positive  indication  of  vehicle  attitude,  terrain 
texture  was  more  readily  discernable  through  observation  of  vehicle  hood  vibrations. 

As  a  result  of  these  observations,  NOSC  has  initiated  studies  to  determine  how  best 
to  present  remote  vehicle  attitude  information  to  the  operator. 

4.4  Head-Coupled  Aiming  of  Sensors.  Head-coupled  aiming  of  driving  sensors 
distinguish  the  TOV  system  from  other  teleoperated  vehicles  systems  intended  for 
military  field  operation.  Operator  pitch  and  yaw  head  orientation  is  sensed  and  used  to 
aim  the  mobility  sensor  platform's  pitch  and  yaw  orientation.  Head-coupled  slaving  of 
the  mobility  sensor  platform  give  the  operator  intuitive  knowledge  of  where  the  video 
cameras  are  aimed  in  three-dimensional  space. 

Head  coupling  of  the  TOV  mobility  system  is  implemented  as  follows.  An 
electromagnetic  sensor  affixed  to  the  TOV  operator's  helmet  senses  operator's  head 
orientation  and  generates  commands  that  are  transmitted  to  the  Remote  Vehicle.  There, 
these  commands  are  used  to  adjust  the  pitch  and  yaw  orientation  of  a  multi-axis, 
motorized,  gimballed  sensor  platform  that  contains  the  stereoscopic  vision  and  binaural 
sensors.  Changes  in  the  remote  visual  aspect,  due  to  sensor  platform  movements,  are 
viewed  by  the  operator  through  miniature  CRT  monitors  attached  to  a  helmet.  Thus, 
operator  pitch/yaw  head  movements  are  designed  to  modify  the  sensor  platform  viewing 
angle  such  that  an  one-to-one  spatial  correspondence  between  head  and  platform  is 
achieved. 

This  TOV  feature  has  several  benefits.  Head-coupled  aiming  control  of  the 
stereoscopic  cameras  promotes  the  remote  presence  sensation.  This  may  result  in 
reduced  training  time  and  easier,  natural  control  of  camera  views.  The  operator  is  able 
to  scan  the  entire  visual  hemisphere  using  simple  head  scans  and  nods,  and  while 
retaining  high  resolution  viewing  in  the  line  of  sight.  This  is  possible  while 
automatically  retaining  spatial  correspondence  of  objects  to  the  vehicle. 

Proprioception,  or  the  awareness  of  neck  and  body  position,  while  locating  objects 
allows  the  operator  to  remember  where  those  objects  lie  in  three  dimensional  space.  An 
additional  benefit  of  head-coupled  aiming  of  sensors  is  that  it  keeps  the  operator's  hands 
and  attention  free  to  tend  to  other  tasks,  such  as  steering  the  vehicle. 

TOV  operators  were  able  to  take  advantage  of  this  feature  in  several  ways.  Most 
operators  are  quick  to  learn  the  concept  of  head-coupled  camera  control.  It  was  observed 
that  operators  use  the  head  coupling  feature  effectively  to  monitor  the  condition  of  the 
RV,  to  scan  the  local  terrain  while  driving,  and  to  fixate  on  auditory  and  visual  targets. 
Although  vehicle  condition  status  reports  are  available  through  the  TOV  panel  mounted 
display,  several  operators  reported  that  It  was  more  comforting  to  see  video 
confirmation  of  shift  lever  position,  brake  pedal  position,  and  other  controls  by  remote 
observation  of  these  controls,  rather  than  status  displays  derived  from  actuator  sensors. 
It  was  also  observed  to  be  very  easy  to  head-aim  the  camera  toward  the  RV  function  of 
interest  since  spatial  position  correspondence  of  objects  were  similar  to  what  one  would 
expect  when  directly  driving  the  vehicle.  While  conducting  remote  driving  trials, 
operators  were  observed  to  use  head  coupling  of  the  cameras  to  scan  the  local  terrain. 
This  is  especially  important  when  making  sharp  turns.  Operators  are  able  to  confidently 
and  accurately  negotiate  turns  because  they  are  at^e  to  view  and  anticipate  where  they 
are  headed.  Head  coupling  also  provides  the  benefit  of  locating  and  turning  to  visual  and 
auditory  targets  with  relative  ease.  Proprioceptive  sensing  allows  operators  to  find 
previously  located  targets  through  correlation  of  object  location  with  operator  neck 


position.  Gross  fracking  of  visual  targets  using  head  coupling  is  maintained  through  head 
movements.  Auditory  tracking  of  targets  is  also  possible  through  the  use  of  the  binaural 
audio  sensors  and  head  coupling. 

Perceptual  cue  conflicts,  attributable  to  the  implementation  of  the  TOV  head 
coupling  scheme,  were  observed  during  testing.  A  combination  of  head  sensing,  platforrii 
actuation,  video,  and  telemetry  delays  contribute  to  the  overall  perception  that  the  image 
lags  rapid  head  motions.  The  perception  that  video  imagery  lags  the  head  motion  has  a 
range  of  effects  on  TOV  operators  and  performance.  This  perceptible  lag  is  usually 
noticeable  by  the  operator  but  is  often  ignored  during  driving  operations.  Some 
operators  are  annoyed  and  even  discomforted  by  the  lag.  More  responsive  head  tracking 
sensors  and  remote  platforms  [9]  may  be  used  to  reduce  this  source  of  discomfort. 

4.5  Head-Mounted  Displays  (HMD)  are  designed  to  be  worn  on  the  operator's  head. 
Since  the  displays  move  with  the  operator's  head,  this  configuration  keeps  the  visual 
display  in  sight  while  allowing  unrestricted  head  movements.  When  used  in  conjunction 
with  head  position  tracking  systems,  HMDs  can  provide  a  natural,  hands  free  method  of 
aiming  video  sensors. 

TOV  HMDs  have  a  pair  of  video  monitors  to  provide  binocular  viewing  using 
separate  left-  and  right-eye  displays.  The  video  monitors  accept  EIA  RS-170A  or  NTSC 
video  signals,  and  convert  them  to  visual  images.  Video  resolution,  overall  video  display 
size  and  weight  are  some  of  the  critical  factors  used  to  evaluate  monitors.  Telescope 
eyepieces  are  used  to  magnify  the  images  from  the  CRT  monitor,  which  are  placed 
directly  in  front  of  the  operator's  eyes.  When  used  with  a  19  mm  diagonal  CRT  image, 
the  apparent  field  of  view  to  the  operator  is  40  degrees.  When  used  with  a  25  mm 
diagonal  CRT  image,  the  apparent  field  of  view  is  55  degrees. 

The  CRT  stereo  alignment  mechanism  secures  the  CRTs  in  front  of  the  operator's 
eyes.  They  are  intended  to  be  aligned  for  full  (100  percent)  binocular  overlap  and  such 
that  the  CRTs  optical  axes  are  parallel  to  each  other.  A  modified  aviator's  night  vision 
system  (AN/AVS-6)  binocular  assembly  is  used  to  provide  these  functions  through  four 


Figure  3.  TOV  Head  Mounted  Display. 


vernier  knob  controls  (up/down,  interocular  sepuration,  in/out,  tilt).  The  helmet,  onto 
which  the  stereo  alignment  mechanism  is  mounted,  is  a  standard  U.  S.  Army  combat 
vehicle  crewman  helmet,  model  DH-132A. 

Two  versions  of  HMDs  were  developed  to  support  TOV  exercises.  (lOj  One 
implementation,  shown  in  figure  3,  provides  high  resolution  (700  horizontal  television 
lines  per  picture  height  (HTVL/PH))  video  displays  with  moderate  FOV  (40  degrees 
horizontal).  The  second  uses  alternate  CRT’s  that  resolves  250  HTVL/PH  with  a 
wider  FOV  (55  degrees  horizontal).  During  operational  evaluation,  the  mobility  system 
camera  lenses  are  changed  to  match  to  the  HMD  FOV.  These  lenses  are  matched  to  the 
display  such  that  the  sensor  and  display  FOVs  are  nearly  equivalent.  For  example,  if  the 
HMD  provides  40  degrees  FOV,  then  the  camera  lens  focal  length  is  selected  to  provide 
40  degrees  FOV.  Table  4.1  lists  HMD  parameters  and  corresponding  camera  lens  focal 
lengths.  The  equivalent  visual  acuity  for  these  vision  systems  was  calculated  and 
verified  through  experimentation.  It  was  estimated  that  visual  acuity  for  the  higher 
resolution  HMD  is  20/95  and  20/155  for  the  lower  resolution  HMD.  By  comparison, 
many  U.  S.  driving  laws  require  corrected  visual  acuity  of  20/40  or  better. 


Table  1.  TOV  HMD  Visual  Acuity. 


Camera  Lens  (focal  length) 
Horizontal  Display  Resolution 
Vertical  Display  Resolution 
Equivalent  Visual  Acuity 


HMD  Apparent  FOV 


40  degrees 
12.5  mm 
700  HTVL/PH 
350  TV  lines 
20/95 


55  degrees 

8  mm 

250  HTVL/PH 
350  TV  lines 
20/155 


The  lack  of  high  visual  acuity  and  the  relatively  narrow  FOV  are  among  the 
common  complaints  with  TOV  drivers.  Operators  desire  higher  image  resolution  to 
perform  more  detailed  inspection  of  targets.  Most  operators  also  want  a  wider  field  of 
view  so  that  an  instantaneous  view  of  the  entire  vehicle's  hood  is  possible.  With  40 
degrees  FOV,  it  is  possible  to  see  the  entire  hood  only  by  scanning  left  and  right.  This 
exercise  is  tedious  when  maneuvering  near  obstacles  since  the  operator  is  required  to 
constantly  scan  back  and  forth  in  order  observe  both  extremes  of  the  vehicle  hood. 
However,  TOV  drivers  indicated  that  head-coupled  scans  were  useful  in  retaining  spatial 
awareness  of  the  vehicle  and  its  obstacles  during  these  maneuvers.  The  55  degrees  FOV 
system  provided  better  peripheral  FOV  at  the  expense  of  lower  image  resolution. 

Overall,  TOV  operators  favored  the  HMD  with  a  moderate  40  degrees  FOV  and  high 
resolution.  They  compensated  for  the  limited  FOV  with  more  head  scans.  The  HMD 
configuration  was  also  extensively  exploited  to  view  the  entire  visual  hemisphere  using 
head  motion. 

Irt  order  to  combine  the  benefits  of  the  moderate  FOV  display  with  high  image 
resolution,  it  is  proposed  that  a  telescopic  camera  be  co-located  with  the  stereo  camera 
pair.  The  operator  would  be  provided  the  capability  of  instantly  switching  between 
viewing  of  the  stereo  pair  and  telescopic  cameras.  The  telescopic  camera  would  be 
aligned  so  that  it  magnifies  the  central  viewable  region  of  the  stereo  pair  image,  and 
provides  a  biocular  view  to  the  operator. 


4 . 6  Alternate  Vision  Systems.  Alternate  vision  system  concepts,  using  head-coupled 
aiming  of  sensors,  have  been  considered  for  UGV  application.  Two  of  the  promising 
approaches  are  the  "virtual  windov/"  and  “panoramic  window’  concepts. 

The  virtual  window  display  is  a  panel  mounted  display  that  uses  operator  head 
position  to  adjust  the  viewing  position  of  the  remote  video  sensors.  The  underlying 
concept  is  to  provide  the  operator  with  a  view  of  the  remote  scene  through  a  simulated 
window.  The  display  represents  the  window  opening  and  the  operator  may  peer  through 
the  window  from  different  viewing  perspectives,  through  horizontal  or  vertical  head 
adjustments. 

During  field  exercises  at  Camp  Pendleton,  a  mock  up  virtual  window  was 
constructed  to  test  this  idea.  A  hole,  the  size  of  a  43  cm  diagonal  CRT  was  cut  out  of  a 
large  cardboard  sheet.  The  cardboard  was  mounted  in  the  driver's  compartment  of  a 
HMMWV  and  the  viewport  was  positioned  to  accommodate  the  vehicle  driver.  Several 
kilometers  of  the  test  course  were  driven  in  this  manner.  For  most  driving  situations, 
the  mock  up  virtual  window  provided  adequate  viewing  coverage.  However,  since  it  was 
not  possible  to  look  more  than  90  degrees  to  each  side  of  the  vehicle's  center, 
approaching  and  crossing  intersections  was  a  challenge. 

The  panoramic  window  display  uses  multiple  video  displays  arranged  in  a  circle 
around  the  operator.  In  one  embodiment,  a  corresponding  circular  array  of  video 
cameras  provide  images  to  the  video  displays.  This  arrangement  provides  instantaneous 
coverage  of  the  entire  360  degrees  FOV.  The  operator  can  view  any  part  of  this  by 
turning  to  look  at  the  video  display  of  interest.  However,  practical  limitations  of  video 
signal  transmission  prevents  simultaneous  real-time  updates  of  all  images.  The 
panoramic  window  concept  requires  transmission  of  only  a  single  video  signal  (or 
stereoscopic  pair).  Images  from  that  signal  are  painted  onto  the  video  monitors  only  in 
the  direction  of  the  operator's  gaze.  The  video  sensor's  line  of  sight  is  slaved  to  the 
orientation  of  the  operator's  head.  This  presents  an  illusion  of  moving  the  viewport 
panoramically  around  the  operator  based  on  head  position. 

Hardware  supporting  the  panoramic  window  concept  has  been  developed  but  has 
not  been  tested  with  UGV  systems. 

5.0  SUMMARY  AND  RECOMMENDATIONS 

A  UGV  remote  vision  system  has  been  developed  incorporating  stereoscopic  vision, 
head-coupled  aiming  of  sensors,  head-mounted  displays,  and  (x>lor  imagery.  This 
implementation  builds  on  NOSC's  past  experience  with  remotely  controlled  vehicles, 
manipulator  work  systems  and  remote  presence  research.  The  vision  system  allowed 
operators  to  remotely  maneuver  the  TOV  RV  over  din  roads  at  velocities  in  excess  of  60 
km/hr  with  typical  driving  speeds  of  25  km/hr.  Furthermore,  the  advanced  vision 
system  features  appeared  to  reduce  operator  training  time,  permitting  system  operation 
after  only  a  few  minutes  of  instruction. 

Field  exercises  of  TOV  systems  identified  areas  for  vision  system  improvement. 
The  lack  of  adequate  RV  attitude  awareness  is  recognized  as  one  of  the  more  serious 
obstacles  to  optimal  system  performance.  Image  resolution  and  sensor  platform 
mechanism  response  are  other  areas  that  need  to  be  evaluated. 

Investigation  of  the  following  vision  system  features,  for  UGV  applications,  is 
recommended: 

1)  stereoscopic  vision; 

2)  color  imagery; 

3)  terrain  slope  perception  cues; 


4)  head-coupled  aiming  of  sensors. 

It  is  believed  that  further  studies  will  confirm  the  benefit  of  these  features  for  the  UGV 
driving  task. 

Stereoscopic  vision  studies  should  be  initiated  to  quantify  the  benefit  of  binocular 
depth  perception  to  the  off-road  driving  task.  It  is  recommended  that  the  studies  include 
the  impact  of  depth  perception  under  conditions  of  degraded  image  quality.  For  example, 
FUR  and  low  light  level  video  imagers  have  limited  sensing  resolution.  While  one  FUR 
image  may  be  unintelligible,  a  FUR  stereo  pair  may  dramatically  enhance  one's 
perception  of  objects  in  view. 

Similarly,  the  benefit  of  color  imagery  should  be  systematically  investigated. 
Color  adds  a  dimension  which  often  cannot  be  recovered  through  black-and-white 
renditions.  It  is  believed  that  target  detection  and  recognition  can  be  improved  using 
color.  It  is  also  thought  to  be  useful  in  the  more  dynamic  task  of  remote  vehicle  driving. 

More  effort  is  required  in  the  area  of  terrain  slope  perception.  Unless  UGV 
operators  have  adequate  awareness  of  the  vehicle's  pitch  and  roll  state,  they  are  in 
danger  of  driving  into  vehicle  roll  over  situations.  Proper  application  of  image  motion 
compensation  techniques  for  attitude  awareness  may  contribute  to  reduced  operator 
fatigue. 

Head-coupled  aiming  of  sensors  is  a  key  component  of  the  TOV  remote  presence 
configuration.  Experiments  to  determine  the  importance  of  hands  free  aim  control, 
spatial  awareness,  and  the  ability  to  scan  a  very  wide  hemispherical  area  should  be 
conducted.  Our  experience  with  TOV  n-j  cates  that  head  aiming  of  sensors  is  highly 
desirable  and  often  indispensable. 

Panel  mounted  driving  display  options  including  virtual  window  concepts  and 
scannable  panel  displays  require  further  development. 
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